Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

Abstract Autograding short textual answers has become much more feasible due to the rise of NLP and increased availability question-answer pairs brought about by a shift online education. performance is still inferior human grading. The statistical black-box nature state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns limiting their practical utility. Furthermore, evaluation autograding typically confined small, monolingual datasets for specific question type. This study uses large dataset consisting 10 million from multiple languages covering diverse fields such as math language, strong variation in answer syntax. We demonstrate effectiveness fine-tuning transformer complex datasets. Our best hyperparameter-tuned model yields an accuracy 86.5%, comparable that are less general tuned type question, subject, language. More importantly, we address trust concerns. By involving humans process, show how improve automatically graded answers, achieving equivalent teaching assistants. also teachers can effectively control errors made system they validate efficiently autograder’s on individual exams close expected performance.